Hazard detection from a camera in a scene with moving shadows

ABSTRACT

Computerized methods are performable by a driver assistance system while the host vehicle is moving. The driver assistance system includes a camera connectible to a processor. First and second image frames are captured from the field of view of the camera. Corresponding image points of the road are tracked from the first image frame to the second image frame. Image motion between the corresponding image points of the road is processed to detect a hazard in the road. The corresponding image points are determined to be of a moving shadow cast on the road to avoid a false positive detection of a hazard in the road or the corresponding image points are determined not to be of a moving shadow cast on the road to verify detection of a hazard in the road.

CROSS REFERENCE TO RELATED APPLICATIONS

The present application is a continuation of U.S. patent applicationSer. No. 15/867,550, filed Jan. 10, 2018, which is a continuation of andclaims priority to U.S. patent application Ser. No. 14/737,522, filedJun. 12, 2015, which claims priority from U.S. provisional patentapplication 62/013,562 filed 18 Jun. 2014 herein by reference. Allreferenced applications are incorporated herein by reference in theirentirety.

BACKGROUND 1. Technical Field

The present invention relates to processing image motion from a movingcamera mounted in a host vehicle for detection of hazards.

2. Description of Related Art

During the last few years camera based driver assistance systems (DAS)have been entering the market; including lane departure warning (LDW),automatic high-beam control (AHC), traffic sign recognition (TSR),forward collision warning (FCW) and pedestrian detection.

A core technology behind forward collision warning (FCW) systems andheadway distance monitoring is detection and class-based recognitionincluding vehicles and pedestrians. A key component of a typical forwardcollision warning (FCW) algorithm is the estimation of distance from acamera and the estimation of scale change from thetime-to-contact/collision (TTC) as disclosed for example in U.S. Pat.No. 7,113,867.

Reference is now made to FIGS. 1 and 2 which illustrate a driverassistance system (DAS) 16 including a camera or image sensor 12 mountedin a vehicle 18. Image sensor 12, imaging a field of view in the forwarddirection provides image frames 15 in real time which are captured by animage processor 30. Processor 30 may be used to process image frames 15simultaneously and/or in parallel to serve a number of driver assistancesystems/applications. By way of example in FIG. 2, image frames 15 areused to serve pedestrian detection 20, lane departure warning (LDW) 21,forward collision warning (FCW) 22. Processor 30 may be used to processimage frames 15 to detect and recognize an image of an object, e.g.vehicle or pedestrian, in the forward field of view of camera 12. Lanedeparture warning (LDW) 21 may provide a warning in the case ofunintentional lane departure. The warning may be given when the vehiclecrosses or is about to cross the lane marker. Driver intention isdetermined based on use of turn signals, change in steering wheel angle,vehicle speed and brake activation. Driver assistance system (DAS) 16also includes real time hazard detection 23 by sensing the verticaldeviation of the contour of the road or deviation from the road planesuch as according to teachings of US patent application publicationUS20150086080. Driver assistance system (DAS) 16 may be implementedusing specific hardware circuitry (not shown) with on board softwareand/or software control algorithms in storage 13. Image sensor 12 may bemonochrome or black-white, i.e. without color separation or image sensor12 may be color sensitive. In some cases, image frames 15 arepartitioned between different driver assistance applications and inother cases the image frames 15 may be shared between the differentdriver assistance applications.

Reference is now made to FIG. 3 which illustrates camera or pinholeprojection which relates a point P (X,Y,Z) in world space Cartesiancoordinates to a point p (x,y) image coordinates on image plane 15 whereX is the horizontal Cartesian coordinate in world space, Y is thevertical Cartesian coordinate in world space and Z is the directionalong the optical axis of camera 12. The origin O of camera projectionis at the pinhole, image plane 15 is in reality is behind the origin atfocal length f with the image inverted. Image plane 15 is shown in theprojection of FIG. 3 in a symmetric position with a non-inverted imagein front of origin O at a distance focal length f. The equations thatfollow, approximate the relation between image coordinates x, y andworld space coordinates X, Y, Z assuming camera or pinhole projection.

$\begin{matrix}{x = {f\frac{X}{Z}}} & (1) \\{y = {f\frac{Y}{Z}}} & (2)\end{matrix}$

The term “homography” as used herein refers to an invertibletransformation from a projective space to itself that maps straightlines to straight lines. In the field of computer vision, two images ofthe same planar surface in space are related by a homography assumingthe pinhole camera model.

Structure-from-Motion (SfM) refers to methods for recoveringthree-dimensional information of a scene that has been projected ontothe back focal plane of a camera. The structural information derivedfrom a SfM algorithm may take the form of a set of projection matrices,one projection matrix per image frame, representing the relationshipbetween a specific two-dimensional point in the image plane and itscorresponding three-dimensional point. SfM algorithms rely on trackingspecific image features from image frame to image frame to determinestructural information concerning the scene.

Reference is now made to FIG. 4, a flow chart showing details ofprocessing of image motion or Structure-from-Motion (SfM) algorithm,according to US patent application publication US20150086080. US patentapplication publication US20150086080 is incorporated herein byreference as if fully set forth herein.

It is assumed that a road can be modeled as an almost planar surface.Thus imaged points of the road move in image space according to ahomography.

In particular, by way of example, for a given camera 12 height (1.25 m),focal length (950 pixels) and vehicle motion between frames (1.58 m), itmay be possible to predict the image motion of selected correspondingpoints on the road plane between the two image frames 15 a and 15 brespectively as host vehicle 18 moves forward. Using a model of thealmost planar surface for the motion of the road points, it is possibleto warp the second image 15 b towards the first image 15 a. Thus, instep 501, image frame 15 b is initially warped into image frame 15 a.(In a similar process, image frame 15 a may be initially warped intoimage frame 15 b).

Instead of trying to find feature points, which would invariably give abias towards strong features such as lane marks and shadows, a fixedgrid of points is used for tracking (step 507). A grid of points isselected (step 503) from a region, e.g. trapezoidal, that roughly mapsup to 15 meters ahead and one lane in width. Points may be spaced every20 pixels in the horizontal (x) direction and 10 pixels in the vertical(y) direction. An alternative would be to randomly select pointsaccording to a particular distribution.

Around each point in image 15 a a patch is located (step 505). The patchmay be 8 pixels in each direction centered around the point resulting ina 17×17 pixel square. The normalized correlation is then computed (e.g.Matlab™ function norm×corr2) for warped image 15 b, where the patchcenter is shifted in the search region. In practical use system 16 mayinclude be a yaw sensor but no pitch sensor and so a tighter searchregion may be used in the x direction rather than in the y direction. Asearch region of (2×4+1) pixels in the x direction may be used and(2×10+1) pixels in the y direction.

The shift which gives the maximum correlation score is found and may befollowed by a refinement search around the best score position with asub-pixel resolution of 0.1 pixels. Invalid tracks may be filtered outat the search stage by picking those points with a score above athreshold (e.g. T=0.7) leaving tracked points 509 as a result oftracking (step 507) and that the reverse tracking from warped image 15 bto image 15 a gives a similar value in the opposite direction. Reversetracking is similar to left-right validation in stereo.

Tracked points 509 as a result of tracking step 507, are fit to ahomography (step 511) using RANdom SAmple Consensus (RANSAC). A number,e.g. four, of points are chosen at random and used to compute thehomography. Points 509 are then transformed using the homography and thenumber of points which are closer than a threshold are counted. Randomlychoosing 4 points and counting the number of points which are closerthan a threshold may repeated many times and the four points that gavethe highest count are retained.

At the end of process 40, the four best points are used to again (step513) transform the points and all the points (inliers) that are closerthan a (possibly different) threshold are used to compute a homographyusing least squares. The rest of the points that are not closer than a(possibly different) threshold are considered outliers.

AI this point in process 40, the number of inliers and their spread inthe warped image give an indication to the success of finding the roadplane model. It is usual to get over 80% inliers and a good fit. Thehomography can then be used to correct the initial alignment for warping(step 501). Correction of the initial alignment can be done byintegrating the correction into the initial warp (step 501) or to do twowarps consecutively. The former is advantageous as it requires only oneinterpolation step and can be performed optionally by matrixmultiplication of the two homography matrices.

After warping image 15 b towards image 15 a to give warped image, usingthe refined warp (step 513), the tracking of points (step 507) may berepeated using a finer grid (e.g. every 5th pixel on every 5th row) andover a wider region of the road. Since the road plane is very wellaligned, a smaller region may be searched over such as 2 pixels in eachdirection, again, with a subpixel search.

Using an image flow analysis between tracked image points 509 asdescribed in US20150086080 or other optical flow analysis algorithms,points on the road have a characteristic positive image flow as hostvehicle 18 moves forward. Positive image flow is defined as flow awayfrom the focus of expansion (FOE) (generally speaking down and outwardsin image frames 15).

Object points in world space above the road plane such as an elevatedsidewalk have an image flow greater than the characteristic image flowof tracked points 509 of the road. Object points in world space belowthe road plane such as sunken manhole covers, have an image flow lessthan the characteristic image flow of tracked points 509 of the road.

Using the image flow analysis between tracked image points 509 asdescribed in US20150086080, image flow of tracked points 509 is comparedwith the expected image flow of the modeled road plane and anydifferences or residual image flow are associated with verticaldeviation in the road. Tracked image points 509 of objects above theroad plane have residual image flow greater than zero and objects belowthe road plane have residual image flow below zero.

BRIEF SUMMARY

Various driver assistance systems mountable in a host vehicle andcomputerized methods performable by the driver assistance systems areprovided for herein while the host vehicle is moving. The driverassistance systems include a camera operatively connectible to aprocessor. First and second image frames are captured from the field ofview of the camera. Corresponding image points of the road are trackedfrom the first image frame to the second image frame. Image motionbetween the corresponding image points of the road is processed todetect a hazard in the road. The corresponding image points aredetermined to be of a moving shadow cast on the road to avoid a falsepositive detection of a hazard in the road and/or the correspondingimage points are determined not to be of a moving shadow cast on theroad to verify detection of a hazard in the road, by at least one of:

(i) hypothesizing that the image motion is consistent with ahypothetical static object in the road, projecting a bottom edge on theroad plane of the hypothesized static object onto the image frames,rejecting the hypothesis that the image motion is caused by a staticobject in the road if there is no strong horizontal texture at thehypothesized projected bottom edge of the static object or if thetexture of hypothesized bottom edge of the static object is changingover the image frames to confirm that the image motion is caused by themoving shadow, or confirming the hypothesis that the image motion iscaused by a static object in the road if there is a strong horizontaltexture at the hypothesized projected bottom edge of the static objectand if the texture of hypothesized bottom edge of the static object isfixed over the image frames;

(ii) recognizing in the image frames images of a moving vehicle andassociating the image motion between the corresponding image points ofthe road with the images of a moving vehicle to determine that the imagemotion between the corresponding image points of the road is due to amoving shadow cast from the moving vehicle. It may be hypothesized thatthat the image motion is due to a hypothetical moving shadow cast fromthe moving vehicle. Respective distances may be computed in the imageframes to the hypothetical moving shadow and the distances to thehypothetical moving shadow may be matched with distances to the movingvehicle to confirm the hypothesis that the image motion is due to themoving shadow cast from the moving vehicle. Alternatively or inaddition, the relative speed between the host vehicle and the movingvehicle may be determined. The image motion of points on the road may becalculated for the relative speed between the host vehicle and themoving vehicle. The image motion of the hypothetical moving shadow maybe measured. The measured image motion of the hypothetical moving shadowmay be compared to the calculated image motion for the relative speedbetween the host vehicle and the moving vehicle and it may be confirmedthat the moving image points are on the moving shadow if the measuredimage motion closely matches the calculated image motion assuming therelative speed between the host vehicle and the detected vehicle.

(iii) associating image motion with a moving shadow when consistent witha static hazard in the road of height greater than a threshold height;

(iv) recognizing images of a lane mark in the road and associating imagemotion with a moving shadow by detecting the image motion superimposedover the images of the lane mark and/or associating image motion with amoving shadow when the image motion is similar on both sides of theimages of the lane mark;

(v) providing location information of an object suspected to cast amoving shadow, finding lines in the image frames between the shadow andthe image of the object and searching along the lines to correlate theimage motion of the object suspected to cast a moving shadow and theimage motion of the shadow;

(vi) verifying that the image motion is due to a moving shadow when theimage motion is not aligned with the focus of expansion or whendirection of the image motion changes sign.

(vii) performing texture analysis on an image of an object suspected tocast a moving shadow to determine if the image of the object has branchtexture and/or leaf texture; and/or

(viii) recognizing an image of a street lamp, verifying that the imagemotion is due to a moving shadow from the light of the street lamp byconstructing lines between the image of the street lamp, the image ofthe tree casting the moving shadow and the moving shadow.

The foregoing and/or other aspects will become apparent from thefollowing detailed description when considered in conjunction with theaccompanying drawing figures.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention is herein described, by way of example only, withreference to the accompanying drawings, wherein:

FIGS. 1 and 2 illustrate a system including a camera or image sensormounted in a vehicle system;

FIG. 3 illustrates camera or pinhole projection;

FIG. 4 illustrates a flow chart showing details of processing of imagemotion or Structure-from-Motion (SfM) algorithm, according to US patentapplication publication US20150086080;

FIGS. 5a, and 5b illustrates road scenes of successive image framescaptured from a camera mounted in a host vehicle, according to featuresof the present invention;

FIGS. 5c and 5d illustrate additional road scenes of successive imageframes captured by from the camera mounted in the host vehicle,according to features of the present invention;

FIG. 6 illustrates a multiple tree texture in an image frame accordingto a feature of the present invention;

FIG. 7a illustrates an image frame showing an oncoming vehicle andtrailer in an adjacent lane creating a moving shadow in the lane of hostvehicle, according to a feature of the present invention;

FIG. 7b shows tracked points on the road plane for shadows of thevehicle and the trailer illustrated in FIG. 7 a;

FIG. 7c which includes the image frame of FIG. 7a and illustratesfurther features of the present invention; and

FIG. 8 includes a flow diagram of a method, according to features of thepresent invention.

DETAILED DESCRIPTION

Reference will now be made in detail to features of the presentinvention, examples of which are illustrated in the accompanyingdrawings, wherein like reference numerals refer to the like elementsthroughout. The features are described below to explain the presentinvention by referring to the figures.

Before explaining features of the invention in detail, it is to beunderstood that the invention is not limited in its application to thedetails of design and the arrangement of the components set forth in thefollowing description or illustrated in the drawings. The invention iscapable of other features or of being practiced or carried out invarious ways. Also, it is to be understood that the phraseology andterminology employed herein is for the purpose of description and shouldnot be regarded as limiting.

By way of introduction, various embodiments of the present invention areuseful to improve hazard detection 23 using structure from motion SfMalgorithms in the presence of moving objects and moving shadows. Underan assumption that the road environment is rigid, image motion of aportion of the road plane may be predicted based on camera projection(equations 1 and 2) and the motion of the host vehicle. Image motiongreater than that of the road plane appears in camera projection as anobstacle in the road with height above the road plane. Image motion lessthan that of the road plane appears in camera projection as a depressionin the road. However, the real road environment is not rigid and thereare moving objects which do not fit a rigid world assumption. The imagemotion of these moving objects/shadows cannot be used to deriveunambiguously height from the road using structure-from-motion (SfM)algorithms such as the teachings of US20150086080 which do notexplicitly account for motion of objects and shadows. System 16algorithm 23 (FIG. 2) which detects hazards based on image motion maygive false positive warnings. The image motion of these objects may warnof hazards that are not there, cause false alarms or worse, falsebraking of vehicle 18. Moving objects often create moving shadows on theroad and these, as will be shown need to be considered. The descriptionthat follows is directed to various solutions to these problems.

Moving objects or shadows may affect the stage of determining areference road plane (according to the teachings of US20150086080).However, image points, which are not moving as image points of the roadplane are expected to move, are detected as outliers in the randomsample consensus (RANSAC) process and are not expected to affect thereference road plane determination. If the outliers are too numerous,detecting of the reference plane will fail because the low inlier countindicates a failure. A threshold may be set if the inlier count dropsbelow the threshold and an invalid flag may be raised. After a fewframes of the inlier count being above the threshold, the referenceplane determination may become valid again as the moving objects and/orshadows are out of the region of interest in images frames 15.

Moving objects normally found in the road environment include vehicles,bicycles and pedestrians. In most cases, these moving objects aredetected by the DAS system 16 using class-based recognition techniquesperformed by forward collision warning FCW 22 and pedestrian detection20 and thus image regions corresponding to these recognized objects maybe masked out and ignored by hazard detection 23 which usesstructure-from-motion techniques.

Objects moving in a general direction results in image flow that doesnot in general match the image flow of the rigid world. Thus, images ofgenerally moving objects may be segmented out as hazards. In general,points on objects moving in the same direction as host vehicle 18 willalso show up as hazards although there will be errors in estimation ofheight H of the candidate object and distance Z to the candidate object.It can be shown that for each height H of the point above the groundthere will be one relative speed at which that point will appear to be astationary point on the road and thus might not be segmented as ahazard. However, there is no one relative speed which “works” for twoobject points of different heights (H). Thus a moving object whichnormally has multiple heights cannot “hide” and is detected as a hazard.

Another issue is moving shadows on the road. The image flow of objectpoints of moving shadows does not correspond to the image flow of theplanar road surface. Thus, there is danger of mistaking moving shadowsfor hazards, e.g. depressions or bumps in the road profile. The firstpriority is to avoid false hazard detection due to moving shadows andthe second priority is to determine the road profile of the road whichinclude moving shadows which may appear to be hazards or bumps in theroad profile.

Reference is now made to FIGS. 5a, and 5b , which show road scenes ofsuccessive image frames 15 captured by processor 30 from camera 12mounted in host vehicle 18, according to features of the presentinvention. Captured image frames 15 a and 15 b in FIGS. 5a and 5brespectively show a tree 50 and its shadow cast on the road surface ashost vehicle 18 advances forward along the road. Tree 50 appears to beunmoving and the shadows of the tree appear to be static in the road ashost vehicle 18 advances from a first position in which image frame 15 aof FIG. 5a is captured to the second position in which image frame 15 bof FIG. 5b is captured. Image motion of the tree shadow which may bemeasured in pixels in x, y directions in image space (Cartesiancoordinate axes are shown in FIG. 5a ) appears identical to image motionof the road plane between respective image frames 15 of FIGS. 5a and 5b. Thus, a structure-from-motion algorithm adapted to detect road hazardswould not detect any hazard from the tree shadows of FIGS. 5a and 5 b.

Reference is now made to FIGS. 5c and 5d , which show additional roadscenes of successive image frames 15 c and 15 d captured by processor 30from camera 12 mounted in host vehicle 18, according to features of thepresent invention. In image frame 15 c of FIG. 5c , tree 50 is bent bywind away from host vehicle 18 and in image frame 15 d of FIG. 5d , tree50 is bent by wind toward from host vehicle 18. Image motion of therespective shadows between image frames 15 c and 15 d is not identicalwith and less than image motion of the road and a structure-from-motionalgorithm may interpret image motion of the moving tree shadow as hazardor depression in the road of some depth. Such a detection would be afalse positive detection. Another example of road shadows caused byshrubs is shown in FIG. 6 in which motion of the shadows due to wind mayresult in false positive detection using Structure-from-Motiontechniques.

Moving shadows from trees are examples of unstructured shadow motion.The direction of motion is not necessarily in the direction of the focusof expansion (FOE) and the direction changes in sign as the leaves andbranches sway in the wind. The following approaches may be used todetermine that the image motion is from a moving shadow, does notrepresent a hazard and a collision warning should be inhibited.

A first approach is to segment out regions where the image flow does notmatch the general image flow direction away from the focus of expansion(FOE) and/or the sign and magnitude of the image flow changes in someapparently random way due to wind.

For moving shadows from sunlight, it is possible with precise date, timeand location information to find one or more lines in image frame 15between the shadow and the candidate image of an object suspected tocreate the shadow. A search along such lines may be performed tocorrelate the image motion of the object casting the shadow and themotion of the shadow. Alternatively, if many image lines from movingpoints suspected as being part of moving shadows all originate from thesame candidate image of an object, then it is likely that the movingpoints are in fact part of from moving shadows being cast by the object.

Shadows at night or twilight may originate from street lamps. Trees orbrush moving under the influence of wind positioned under or near thestreet lamps may generate moving shadows which may appear to be hazardsusing a structure-from-motion algorithm. Images of street lamps may berecognized by being aligned along lines at the side of the road whichmeet at the focus of expansion and have an image flow characteristic ofstreet lamps in which the vertical image coordinate y moves according toequation 2 above with Y being the difference between height of thestreet lamp and the camera height. Lines may be drawn between the shadowpoints, the image of the tree casting the shadow and the image of thestreet lamp which are a strong indication that the moving shadows arenot hazards in the road.

In addition texture analysis may be performed on the candidate image tosee if the candidate image has leaf and/or branch texture. Textureanalysis may also be performed in the texture on the road. Even thoughtree shadows may extend to a few meters on the road, due toforeshortening the vertical extent of the shadow in the image is quitesmall. If the moving shadows are large in vertical image coordinate yand the shadows likely originate from many trees and the motion is notexpected to be uniform over the entire shadow. An example of multipletree texture on the road surface is shown in FIG. 6.

Reference is now made again to FIGS. 5c and 5d . A lane mark may bedetected using an output from lane departure warning LDW 21 algorithm.The detected lane mark leads into a a moving shadow (of tree 50) inwhich image motion is detected. Moreover, as shown in FIGS. 5c and 5dthe shadow moves over different portion of a broken lane marker. Theseare strong indications that the moving image region is a moving shadowand not a hazard.

Thus, it is possible with various methods to distinguish between imageflow of a moving object or static hazard in the road, from image flowcreated by moving shadows of trees or bushes.

Reference is now made to FIG. 7a which includes an image frame 15showing an oncoming vehicle and trailer in an adjacent lane creating amoving shadow in the lane of host vehicle 18, illustrating a feature ofthe present invention. The shadow of the oncoming vehicle in theadjacent lane as shown often appears as a dark horizontal strip movingtowards host vehicle 18, the closer (bottom) edge of the shadow and thefarther (upper) edge may be considered. A vehicle with a complex shapemight produce more than two horizontal edges (in this example two pairsof moving edges are visible). The image motion of the shadows on theroad may correspond to a static object of a certain height H anddistance Z. If tracked over multiple frames 15, height H and distance Zremain consistent and appear to be a hazard of height H in the road to astructure-from-motion algorithm. Since the moving shadow is movingtowards host vehicle 18 a structure-from-motion algorithm mayincorrectly interpret the moving shadow to be a hazard of finite heightH above the road plane.

Specifically, reference is now also made to FIG. 7b which shows trackedpoints on the road plane, according to teachings of US20150086080 forthe respective shadows of the vehicle and the trailer illustrated inFIG. 7a . The tracked points marked with triangles near the shadowregions of the oncoming vehicle and its trailer fit the road plane modelbut have strong residual motion. Thus the shadows associated with theoncoming vehicle and trailer appear as false hazards according the theteachings of US20150086080.

The discussion that follows of methods for differentiating between amoving shadow and a real hazard is presented using vehicles as anexample but may be applied equally to pedestrians.

Differentiation between a moving shadow and a real hazard may beaccomplished according to different embodiments of the present inventionas follows:

-   -   If the oncoming vehicle is traveling at a similar speed as host        vehicle 18, the shadow image flow is double the image flow of        the road plane and the shadow appears as a static object of half        camera height or H/2. Shadows from oncoming vehicles may be        rejected as hazards by filtering out apparent objects of height        above a threshold, such as 0.5 meters for example.    -   Lane detection algorithms included in lane departure warning 21        may be used to detect lane marks and individual lane mark        segments of from image frames 15. Reference is made to FIG. 7c        which illustrates features of the present invention. LDW 21        detected a solid lane mark to the right of host vehicle 18 and        lead vehicle 64 as shown with a long line with double arrows. To        the left of host vehicle 18 are three detected lane mark        segments which indicate the middle of the road. If such a lane        mark segment passes through a moving image patch, the moving        image patch is likely a moving shadow and not a hazard.        Furthermore, if the image motion is similar on both sides of the        lane mark then there is an even stronger indication that the        image patch is a shadow of an oncoming vehicle 26.    -   As noted above, the moving shadow may appear as a static object        protruding from the road and it is possible to verify that the        moving shadow is not in fact a real object. The bottom of such        an hypothesized object may be projected onto the image. At that        location in the image significant horizontal texture such as an        edge is expected and the motion of that texture should        correspond to the image motion of the road. Or in the        terminology of US20150086080 the residual image motion should be        zero. Thus the upper edge of the moving shadow will be rejected        because of the texture below it, at the hypothesized bottom        location is not stationary. The lower edge of the shadow is        rejected because, in most cases, there is no strong horizontal        texture at the projected ground point.

A hazard has strong horizontal texture so the patch should have asignificant horizontal edge. An edge detection may be performed and thenumber of horizontal edge points in a patch and in each row of the patchmay be counted. A good horizontal edge should have a cluster ofhorizontal edge points in one or two adjacent rows. The patch used tocount edge points can have more rows and/or columns than the patch usedfor tracking.

A hazard should have a horizontal edge both at the top and where ittouches the road. The edge on the road should have no residual flowassociated with it.

-   1. For each candidate grid point compute the distance (Z) and height    (H).-   2. Compute image height (h) of the hazard candidate: h=fH/Z.-   3. Search a region centered h pixels below the candidate grid point    for a significant horizontal edge.-   4. Track region to see if residual motion is stationary.-   5. Residual motion (image motion not attributable to image motion of    the road plane) and strength of edge are features that can be used    to verify the candidate.    -   Conversely, the edge in the image can be suspected as being a        moving shadow of a vehicle and this hypothesis may be tested        using DAS system 16 running vehicle detection to see if it is        associated with a detected vehicle. Reference is still made to        FIG. 7c which illustrates further features of the present        invention. Driver assistance systems such as vehicle detection        included in forward collision warning FCW 22 may be used to        detect vehicles associated with moving image patches. FIG. 7c        shows the typical output of system 16. In FIG. 7c an oncoming        vehicle 26 is shown as detected and also a lead vehicle 64 is        shown as detected. Correlated changes in respective image        coordinates of an image of vehicle and of a moving image patch        of the road may indicate that the moving image patch is an image        of a moving shadow cast by the vehicle and not a real hazard in        the road.

Points that exhibit significant residual flow (image flow notattributable to a planar road) are suspect hazard points. However,according to a second hypothesis the suspect hazard points might bemoving shadows on the planar surface. Using the ground plane constraint,the distance to the hypothesized shadow can be computed in the twoimages: Z_(i)=fH/(y_(i)−y0_(i)), where y_(i), i=1, 2 is the image rowlocation of the shadow and y0_(i), i=1, 2 is the horizon location, orvanishing row of the road plane, in that image. The hypothesized shadowcomputed in the two images gives distance Z and relative speed and canbe used to match up with detected targets (vehicles) that have similardistance and relative speed. If the hypothesized shadow matches indistance and/or relative speed to a detected target, the hypothesizedshadow is accepted and a hazard warning is inhibited.

An association can make use of the sun location and shadow direction toassociate only with targets to the left or right of the hypothesizedshadow. The target list can come from vehicle detection performed usingcamera 12 or from a second camera with a different, often wider, fieldof view. The targets can come also from Radar or Lidar. The targetvehicle might have left the field of view (FOV) before the shadow is inthe region of the road where hazard detection is being performed. Thetargets vehicle's presence can be maintained till the previouslymeasured distance and speed indicate that the target vehicle is too farout of range to affect the scene.

In summary, reference is now made to FIG. 8, which includes a flowdiagram illustrating a method according to features of the presentinvention.

Image frames 15 are captured (step 801) from the field of view of camera12. Corresponding image points of the road are tracked (step 803) fromimage frame 15 to image frame 15. Image motion between the correspondingimage points of the road is processed (step 805) to detect a hazard inthe road. If a moving shadow is cast on the road, it is determined (step807) that the corresponding image points are of the moving shadow toavoid a false positive detection of a hazard in the road. Alternatively,or in addition it may be determined (step 809) that the correspondingimage points are not of a moving shadow cast on the road to verifydetection of a hazard in the road.

The term “warping” as used herein refers to a transform from image spaceto image space.

The term “corresponding” as used herein refers to matching image pointsin different image frames which are found to be of the same objectpoint.

The term “image texture” or “texture” as used herein refers tointensity, brightness or grayscale changes along a line or curve in animage frame.

The term “focus of expansion” (FOE) as used herein refers to a point inthe image flow from which all image motion appears to emanate.

The indefinite articles “a”, “an” is used herein, such as “an image” hasthe meaning of “one or more” that is “one or more images”.

Although selected features of the present invention have been shown anddescribed, it is to be understood the present invention is not limitedto the described features.

What is claimed is:
 1. A computerized method for processing imagescaptured by a camera mountable in a host vehicle while the host vehicleis moving, the method comprising: capturing a first image frame and asecond image frame from a field of view of the camera; trackingcorresponding image points from the first image frame to the secondimage frame; detecting, from the tracked corresponding image points andbased on a comparison of the first image frame to the second imageframe, a plurality of suspect hazard image points; determining motionbetween the plurality of suspect hazard image points of a road;recognizing a lane mark in the road in the first and second imageframes; detecting that the motion is superimposed over the lane mark inthe first and second image frames; and in response to detecting that themotion is superimposed over the lane mark in the first and second imageframes, associating the motion with a moving shadow.
 2. The method ofclaim 1, further comprising: hypothesizing that the motion is consistentwith a hypothetical static object in the road; projecting a bottom edgeon the road plane of the hypothesized static object onto the first andsecond image frames; rejecting the hypothesis that the motion is causedby a static object in the road and confirming that the motion is notcaused by a hazard if there is no strong horizontal texture at thehypothesized projected bottom edge of the static object or if thetexture of hypothesized bottom edge of the static object is changingover the first and second image frames or confirming the hypothesis thatthe motion is caused by a static object in the road if there is a stronghorizontal texture at the hypothesized projected bottom edge of thestatic object and if the texture of hypothesized bottom edge of thestatic object is fixed over the first and second image frames.
 3. Themethod of claim 1, further comprising: recognizing in the image frames,images of a moving vehicle; associating the motion with the images of amoving vehicle; and determining the motion is not due to a hazard. 4.The method of claim 3, further comprising: hypothesizing that the motionis clue to a hypothetical moving shadow cast from the moving vehicle;computing respective distances to the hypothetical moving shadow;matching the distances to the hypothetical moving shadow with distancesto the moving vehicle; and confirming the hypothesis that the imagemotion is due to the moving shadow cast from the moving vehicle.
 5. Themethod of claim 3, further comprising: hypothesizing that the motion isdue to a hypothetical moving shadow cast from the moving vehicle;determining relative speed between the host vehicle to the movingvehicle; calculating motion of points on the road for the relative speedbetween the host vehicle and the moving vehicle; measuring motion of thehypothetical moving shadow; comparing the measured motion of thehypothetical moving shadow to the calculated motion for the relativespeed between the host vehicle and the moving vehicle; and confirmingthe hypothesis that the moving image points are of a moving shadow castfrom the moving vehicle if the measured motion matches the calculatedmotion.
 6. The method of claim 1, further comprising: associating themotion with a moving shadow when consistent with a static hazard in theroad of height greater than a threshold height.
 7. The method of claim1, further comprising: associating the motion with a moving shadow whenthe motion is similar on both sides of the images of the lane mark. 8.The method of claim 1, further comprising: providing locationinformation of an object suspected to cast a moving shadow; finding aplurality of lines in the first and second image frames between theshadow and the image of the object; and searching along the lines tocorrelate motion of the object and motion of the shadow.
 9. The methodof claim 1, further comprising: performing texture analysis on an imageof an object suspected to cast a moving shadow; and determining if theimage of the object has at least one texture selected from a groupconsisting of: branch texture and leaf texture.
 10. The method of claim1, further comprising: recognizing an image of a street lamp; verifyingthat the motion is due to a moving shadow from a light of the streetlamp by constructing lines between the image of the street lamp, animage of the tree casting the moving shadow, and the moving shadow. 11.A system mountable in a host vehicle, the system including a cameraoperatively connectible to a processor, the processor of the systemconfigured, while the host vehicle is moving, to: capture a first imageframe and a second image frame from a field of view of the camera; trackcorresponding image points from the first image frame to the secondimage frame; detect, from the tracked corresponding image points andbased on a comparison of the first image frame to the second imageframe, a plurality of suspect hazard image points; determine motionbetween the plurality of suspect hazard image points of a road;recognize a lane mark in the road in the first and second image frames;detect that the motion is superimposed over the lane mark in the firstand second image frames; and in response to detecting that the motion issuperimposed over the lane mark in the first and second image frames,associate the motion with a moving shadow.
 12. The system of claim 11,wherein the processor is further configured to: hypothesize that themotion is consistent with a hypothetical static object in the road;project a bottom edge on the road plane of the hypothesized staticobject onto the first and second image frames; reject the hypothesisthat the image motion is caused by a static object in the road andconfirm that the motion is not caused by a hazard if there is no stronghorizontal texture at the hypothesized projected bottom edge of thestatic object or if the texture of hypothesized bottom edge of thestatic object changes over the first and second image frames.
 13. Thesystem of claim 12, wherein the processor is further configured to:recognize in the image frames, images of a moving vehicle; associate themotion with the images of a moving vehicle; and determine that themotion is due to a moving shadow cast from the moving vehicle.
 14. Thesystem of claim 11, wherein the processor is further configured to:provide location information of an object suspected to cast a movingshadow; find a plurality of lines in the first and second image framesbetween the shadow and the image of the object; and search along thelines to correlate motion of the object and motion of the shadow. 15.The system of claim 11, wherein the processor is further configured to:perform texture analysis on an image of an object suspected to cast amoving shadow; and determine if the image of the object has at least onetexture selected from a group consisting of: branch texture and leaftexture.
 16. The system of claim 11, wherein the processor is furtherconfigured to: recognize an image of a street lamp; and verify that themotion is due to a moving shadow from a light of the street lamp byconstructing lines between the image of the street lamp, an image of thetree casting the moving shadow, and the moving shadow.